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Summary, Findings, and Conclusions 

1. Virginia Polytechnic Institute and State University (Virginia Tech) is located on a 2,600- 
acre site in the town of Blacksburg in the southeastern part of the Commonwealth of 
Virginia. The university offers approximately 200 degree programs from the bachelor's 
through doctoral level to 25,000 students. Virginia Tech draws students from through- 
out the nation and the world. Approximately 73 percent are from the Commonwealth. 
Almost all students live on or adjacent to the campus. 

2. As a way to focus upon the need for a university-wide strategy to leverage instructional 
technology to achieve its strategic mission, the Provost and the Vice President for Infor- 
mation Systems began in 1993 the implementation of an "Instructional Development 
Initiative." This large-scale effort was designed to invest in faculty (through a component 
program, the Faculty Development Initiative - FDI) by providing them an opportunity to 
rethink their teaching and explore the potential of (electronic) instructional technology 
for improving the effectiveness of the teaching-learning process. 

3. Faculty who participated in the FDI workshops have been provided access to multimedia 
computers with high speed network connections. The campus has an ongoing program 
to equip classrooms and upgrade computer labs with sufficient computers to provide 
adequate access for students enrolling in the redesigned courses who do not have their 
own computers and network access. 

4. Two professors who participated in the workshops restructured the courses that are 
the basis of this study: Philosophy 1204, Knowledge and Reality, taught by Dr. Valerie 
Hardcastle and Biology 2604, Introductory Microbiology, taught by Dr. G. William 
Claus. Data from these courses as taught in spring term 1997 are the primary basis for 
the evaluation of learning outcomes. 
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Benefits 



Learning Outcomes in the Philosophy Course 

5. Dr. Hardcastle redesigned her course to simulate small group discussion sessions which 
were previously face-to-face. The redesign involved use of a combination of synchro- 
nous and asynchronous methods facilitated by emerging instructional technology 
applications. The redesigned course meets as a group only once per week for a regular 
lecture which primarily consists of an overview of the study topic for the week. 

6. In the spring semester 1997, one section of the philosophy course was taught by 
Dr. Hardcastle in the "interactive web-based mode." The other section was taught 
by another faculty member in a more traditional large lecture/small discussion format. 

7. Students in both sections of the course were assigned to write a persuasive essay. Forty 
essays were randomly selected from each section. These essays were scored and graded 
by three readers (graduate students in philosophy) from another university. 

8. The evidence regarding learning outcomes is somewhat ambiguous. Students in the 
web-based course scored better than the lecture section on eight of sixteen specific 
essay scoring criteria deemed important to philosophical discourse, there was no sig- 
nificant difference between the two sections on seven of the criteria, and the lecture 
section scored significantly better on only one criterion. Based upon these results, the 
web-based section's performance on the essay was superior. This result is consistent 
with the increased writing and interaction regarding the writing that was built into 
the course redesign, i.e., that increased practice and review should improve perfor- 
mance. Even though the web-based group showed superior performance based upon 
the sixteen criteria, letter grades assigned by the readers to this particular essay showed 
no significant difference between the two groups. 

9. In terms of responses to the student survey, there is evidence that student behavior 
was affected to the extent that over half had increased their use of the Internet and 
that over three-quarters better appreciated discussion, peer review and the value of 
informal writing as a component of the learning process (both of these results may 
indicate other learning outcomes that will be very important for students living in 
an increasingly digitized work and social environment). Less than a fourth of the 
student respondents felt there was too much use of the computer in the course. 

10. Advantages most frequently mentioned by students in the web-based section 
were better access to course materials and convenience and flexibility in study- 
ing. The disadvantages most frequently mentioned were technical problems and 
the impersonal nature of the technology. 



Learning Outcomes in the Microbiology Course 

11. Biology 2604 is a sophomore level introductory microbiology course that enrolls 
approximately 200 students per semester. It is taught once each semester by Dr. G. William 
Claus who has been responsible for the course for several years. The class meets three 
times per week in a large lecture hall. There is an associated microbiology laboratory 
course, and, although it is not required, most students take the lab concurrently with the 
lecture course. 
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12. With the help of a grant from the Alfred P. Sloan Foundation, Dr. Claus and a development 
team in Virginia Tech's Educational Technologies Department restructured the course offer- 
ing. The goals of the restructuring were to improve student access to course related materials 
and to improve communications between students and faculty and among students. 

13. The changes took the form of enhancements to the lecture presentation, network 
access to all course materials, and new communications capabilities. All overhead 
transparencies were converted to PowerPoint slides which Dr. Claus used in lecture. 
These materials were in turn converted to files that students could access from the 
web site. An elaborate course web site was created that allowed students to access class 
announcements, frequently asked questions, course documents (e.g., syllabi), color 
slides, printable black and white slides, practice exams, live communication links to 
the professor and the teaching assistant (electronic office hours - "chat room"), direct 
e-mail to faculty, asynchronous discussions with other students ("message boards"), 
and links to other web sites related to biology and microbiology. 

14. Although a statistical test for differences in learning outcomes based upon course 
grade comparisons over time was not performed, the evidence strongly suggests there 
is no difference in either mean grades or the grade distributions for versions of the 
course offered before and after the course redesign. The grades, themselves, provide 
no evidence that learning outcomes have changed as a result of the introduction of 
the network technology. (It is worth reiterating that Dr. Claus’s objective in the course 
transformation was not to change the grades but to improve communications and 
increase student access to course materials.) 

15. The student survey responses clearly indicate that the course web site has improved 
student access to course materials, which was one of the primary objectives of the 
course redesign. 

16. The survey responses indicate, in terms both of activities and attitude changes, that 
students are learning to use the computer and the Internet as tools to assist them in 
their studies. Such results will have direct application in their careers after graduation. 



Institutional Renewal 

17. The Instructional Development Initiative is a large-scale effort to invest in the faculty 
of Virginia Tech by providing them with the opportunity to rethink their teaching 
and to explore the potential of instructional technology for improving the effective- 
ness of the teaching-learning process. This initiative is based upon the realization 
that digital technology is causing profound changes in the way people communicate 
with each other and gain access to various types of information. These changes have 
important implications for the curriculum both in terms of how the technology ap- 
plies in particular disciplines and in terms of how courses can be taught. Students 
must be educated about these applications in order to function in society and the 
labor market. 

18. Virginia Tech has met this responsibility with the Instructional Development Initiative 
and the Faculty Development Institute (FDI). The FDI provides both opportunities and 
incentives for the faculty, the campus's most valuable resource, to learn more about the 
application of the technology as it applies to their teaching activities and to work with 
each other in sharing ideas and experiences about the instructional applications. 
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19. This investment in human capital has been augmented by providing faculty with the 
appropriate desk-top computing equipment and network connections. It also includes 
a large-scale effort to provide network access, specialized computer labs, and support 
for students; classroom upgrades and equipment in classrooms to make them suitable 
for computer presentation and network access. 

20. Many "graduates" of the FD1 have continued to develop their courses and contribute to 
an exciting atmosphere that encourages change and innovation. Dr. Claus, in particular, 
attributes a renewal in his teaching to his experience in the FDI and subsequent support 
from the Department of Educational Technologies, which manages the FDI workshops. 



Student Access 

21. Virginia Tech's emphasis upon the use of the technology in the Instructional 
Development Initiative has been to improve communications and access to course 
materials for on-campus students. The Initiative is broadly conceived, however, and is 
providing the basis for an expansion of Virginia Tech's distance learning activities. 



Costs 

22. The costs of restructuring these courses include the one-time costs of the FDI workshops 
and the course conversions. Providing workstations for faculty and students is a major 
ongoing cost. 

23. For courses that enroll 100-200 students per term, course costs, including FDI, 
course conversion, and workstations for faculty and students, the estimated costs 
are $4,200-$6,000. On a per student enrolled in one course basis the costs are 
$30-$42. 

24. If 25 percent of a student's entire degree program were to be taken in these restructured 
courses, the cost per student would increase by $331-$465. If twenty-five percent of 
the university's courses were restructured, its annual expenditures on these activities 
would increase $1.7-$2.4 million. 
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Background and Context 



Virginia Polytechnic Institute and State University (Virginia Tech) was founded as a land 
grant college in 1872. The university occupies a 2,600-acre site in the town of Blacksburg, 
Virginia, in the southeastern part of the Commonwealth about 40 miles southeast of 
Roanoake. The University offers approximately 200 degree programs from the bachelor's 
through doctoral level to 25,000 students. 

Virginia Tech draws students from throughout the nation and the world. Approximately 73 
percent are from the Commonwealth. The student body is residential; almost all students live 
on or adjacent to the campus. 

Blacksburg is a rural community with a population of 16,000 (not counting Virginia Tech's 
students). It is the site of the Blacksburg Electronic Village (BEV), a joint project of the com- 
munity, Virginia Tech, and Bell Atlantic, to provide access to high speed communications 
throughout the community. 

Even though it is a residential campus, leaders at Virginia Tech have a strong interest in 
distance learning. The institution has been offering distance learning programming for more 
than ten years. The Faculty Development Institute (FDI) includes a track on developing syn- 
chronous and asynchronous instructional delivery modes. It is clear that applications of digi- 
tal electronic technology have the potential for changing delivery modes. There was also the 
need to ensure that students learn how the technology works and how to use it as it applies to 
their fields of study and to the evolving workplace environments. 



The Instructional Development Initiative 1 

As a way to focus upon the need for a university-wide strategy to leverage instructional 
technology to achieve its strategic mission, the Provost and the Vice President for Information 
Systems began in 1993 the implementation of an "Instructional Development Initiative." 
This large scale effort was designed to invest in faculty by providing them an opportunity 
to rethink their teaching and explore the potential of (electronic) instructional technology 
for improving the effectiveness of the teaching-learning process. The Initiative led to the 
Faculty Development Institute which was designed to provide all 1,500 faculty at Virginia 
Tech an opportunity to participate over a four-year period in intensive summer workshops 
centered on the integration of instructional technology with the curriculum. 

Responses to the workshops have been positive with the consequence that, in parallel, with 
the evolution of the technology itself, the content of the workshops has evolved from devel- 
opment of basic computer skills to much more sophisticated topics in the application of 
instructional design principles to course development activities and learning modules appro- 
priate for specific disciplines. In addition to the summer workshops, a series of shorter work- 
shops on various specific topics was conducted during the academic year. The interests of 
faculty have also shifted toward more emphasis upon producing materials that can be used 
by students enrolled in their courses. As part of the evolution of this strategy for leveraging 
instructional technology, twenty network-based courses were offered this summer to students 
from Virginia Tech who moved home to work over the summer. 



1 The following materials have been adapted from Virginia Tech, Instructional Development 
Initiative, August 1996. 
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Faculty who participated in the FDI workshops have been provided access to multimedia 
computers with high speed network connections. The campus has an ongoing program to 
equip classrooms and upgrade computer labs with sufficient computers to provide adequate 
access for students enrolling in the redesigned courses who do not have their own computers 
and network access. Virginia Tech has also been selected as a New Media Center, part of a 
national nonprofit corporation developed to assist campuses with the use of the new media 
in teaching and learning by means of a national network for disseminating information about 
the technology and its implementation in teaching. Virginia Tech has implemented a com- 
puter requirement for incoming freshmen for fall semester 1998. This was a faculty-driven 
initiative designed to ensure that all students had the appropriate resources to access course 
materials and Internet resources. 
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Benefit Estimates 



Evaluations of Two Restructured Courses 



Two professors who participated in the workshops restructured the courses that are the basis of 
this study: Philosophy 1204, Knowledge and Reality, taught by Dr. Valerie Hardcastle and Biol- 
ogy 2604, Introductory Microbiology, taught by Dr. G. William Claus. Data from these courses 
as taught in spring 1997 are the primary basis for the evaluation of learning outcomes. 

Knowledge and Reality 

Philosophy 1204 enrolls approximately 400 students per semester in two sections, which 
are ordinarily taught in large lecture halls. It is a freshman level course, but because it 
satisfies one of the university's core humanities requirements, it enrolls a range of stu- 
dents from freshman to senior level, very few of whom are philosophy or humanities 
majors. Table 1 provides demographic data on the 210 students enrolled in Dr. Hardcastle's 
section of the course. Although some attrition occurred during the early weeks of the 
class, students who dropped were replaced by an equal number of adds, resulting in a 
stable enrollment throughout the semester. 



Table 1 -Class Profile for Dr. Hardcastle's Section of Philosophy 1204 



Class Level 




Percent 




Freshman 


43.5% 




Sophomore 


36.7 




Junior 


10.8 


Senior and Postbaccalaureate 


8.8 


Gender 




Male 


58.5% 




Female 


41.5 


Ethnicity 




Caucasian 


87.0% 




Asian 


6.8 




Latino 


2.1 




Other 


4.1 


Percent Owning Computer 




83.8% 
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Course Redesign - The Interactive Web-based Section 

As a result of significant budget cuts in the early 1990s, departments across campus were 
required to move to larger enrollment courses which were substituted for lower enrollment 
discussion sections. The Department of Philosophy is an example of this restructuring required 
by budget reductions. 

Faced with the increase in class size, Dr. Valerie Hardcastle, a member of the philosophy 
faculty, redesigned her course to simulate small group discussion sessions which were previ- 
ously face-to-face. Using a combination of synchronous and asynchronous methods facili- 
tated by emerging instructional technology applications, she implemented a class web page 
that provides course news and related information, and has extensive interactive communi- 
cation capabilities. As part of the Philosophy 1204 section, she requires students to engage in 
a variety of on-line discourse modes using both live chat sessions and asynchronous methods 
(e.g., a moderated version of a bulletin board). In addition to the assigned readings, several 
"workshops" on various topics in philosophy are offered each week. Students are required to 
participate in 14 of these workshops per semester. 

Some of the discussion sections require on-line discussion and written responses to philosophical 
writings, movies and short stories. Responses can consist of live computer-mediated communica- 
tion (using Daedalus software in a computer lab), synchronous chat sessions (over the web using 
NetForms software), asynchronous discussion postings (also web-based), homework assignments 
submitted over the web, and essays written and submitted in hard copy to the instructor. The 
redesigned course meets as a group only once per week for a regular lecture which primarily 
consists of an overview of the study topic for the week. 

The primary difference in the designs of the new and old course involves what students do to 
fulfill their requirements. In the traditional approach, students read the text, attend class 
frequently and take extensive notes from the instructor's presentation, then write a few (2-3) 
long and polished papers during the semester. In the redesigned course, students also read 
the text, but they engage in much more writing (both informal and formal) in a variety of 
venues (i.e., on-line and asynchronous modes), receive more frequent challenges and critiques 
of their thoughts and assertions, and are required to describe, discuss and debate philosophical 
ideas from a variety of sources, e.g., movies, short stories, as well as philosophical texts. They 
receive much less of their information directly from the professor. 

The Comparative Study 

After teaching the class using these methods for several semesters, she did a comparative 
study of her students' writing with a section of the same class that was not using these 
teaching methods. The traditional class emphasizes finished papers written to be read and 
graded by a graduate teaching assistant while students in the web-based section wrote more 
frequently. Because much of the writing consists of interactive discourse, it is reviewed 
more frequently by peers, by the teaching assistants, and by the professor. In order to test 
the efficacy of this on-line interactive approach compared to the traditional lecture/discussion 
mode, Dr. Hardcastle designed a study to compare performance on an essay assignment 
written both by her students and by students in the traditional section. 

In the spring semester 1997, one section of the course was taught by Dr. Hardcastle, in the 
"interactive web-based mode." The other section was taught by another faculty member in 
a more traditional large lecture/small discussion format. In the "traditional" section, the 
entire class meets twice per week for a regular lecture, and once per week in small discussion 
sections (35 students) led by graduate teaching assistants. 
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In the tenth week of the semester, students in both sections of the course were assigned to 
write a persuasive essay on the same topic. Forty essays were randomly selected from each 
section. These essays were scored and graded by three readers (graduate students in philoso- 
phy) from another university. Each reader read every essay. The readers did not know student 
names or the section in which the student was enrolled. The readers used a ten-point scale to 
rate each essay on 16 criteria all deemed important in terms of philosophical discourse (the 
criteria are listed in Table 2 below). 

At a recent internal conference on instructional technology, Dr. Hardcastle summarized the 
point of the study in this way: 

I should stress that the reason we were looking at this [essay comparison], as opposed to 
some sort of content-based question or student evaluation is because the bread and 
butter in philosophy is whether students can write a successful, sharp, analytical paper. 
Everything else is icing. I don't really care whether they can remember two weeks after 
the course who Descartes was — my apologies to the historians of philosophy. I don't 
care if they can remember anything from the course as long as they can, when con- 
fronted with an issue, be able to articulate their views sharply and defend them with 
rational argumentation. (September 1997) 

A multivariate analysis of variance was performed on the essay scores. It showed significant 
differences, and therefore a univariate analysis of variance (ANOVA) was conducted between 
all items. The results of that analysis are shown in Table 2. 

There was no statistically significant difference between the two groups on seven criteria. The 
lecture section scored better than the web-based section on one criterion (p < .05), and the web- 
based section scored better than the lecture section on eight criteria (four criteria at p < .10 and 
four at p < .05). The web-based section appears to have outperformed the lecture section on 
the basis of the individual scoring criteria. 

The readers were also asked to assign overall grades to the essays. The web-based section's 
average grade was slightly higher, a low "B" (mean grade = 4.74 2 ) compared to a "B-" (mean 
grade = 5.03) for the traditional section. There was no significant difference (p < .10) between 
the mean essay grades for this particular essay for these two sections of the course. Even with 
what appears to be a superior performance based upon the criteria in Table 2, there was not a 
significant difference in the overall grades assigned to the essay. 



2 Calculated on a scale where A = 1 and D- (or less) = 10. 
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Table 2-Comparison of Essay Scores for Web-based and Lecture Sections 
of Philosophy 1204 - Spring Term 1997 



Mean scores of the two sections were not 


Significance 


statistically different (nsd at the p< 0.10 level) 


level 


Scoring Criterion: 


This essay contains a thesis statement or conclusion.* (1) 


nsd 


The thesis or conclusion is clearly stated.* (2) 


nsd 


The reasons or arguments are relevant to the thesis or conclusion.** 


(4) nsd 


No reasons are anachronistic.* (6) 


nsd 


The reasons or arguments are cogent.** (7) 


nsd 


This essay's organization contributes to its readability.** (13) 


nsd 


This essay contains no extraneous material.* (15) 


nsd 


Mean scores of web-based sections were 


Significance 


less than the score of the lecture section 


level 


Scoring Criterion: 


This essay is succinctly written. (16) 


p < .05 


Mean scores of web-based sections were 


Significance 


more than the score of the lecture section 


level 


Scoring Criterion: 


No reasons or premises are missing. (5) 


p< .10 


This essay goes beyond simply stating the obvious. (9) 

The author of this essay understands the philosophical views 


p< .10 


described in the paper. (10) 


p< .10 


This essay's organization contributes to the force of the argument. (12) p < .10 
This essay contains reasons or arguments for supporting the 


thesis or conclusion. (3) 


p < .05 


This essay contains no factual errors. (8) 

This essay exhibits sensitivity to counter-arguments or 


p < .05 


counter-examples. (11) 

No errors in language or usage obscure the meaning of the 


p < .05 


sentences. (14) 


p < .05 



* The mean score of the web-based section was less than the mean score of the lecture 
section. 

**The mean score of the web-based section was greater than the mean score of the lecture 
section. 
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Student Surveys 

During the spring 1997 semester, a survey was administered to the students enrolled in 
Dr. Hardcastle's section of the course. The questions were concerned with the students' 
reaction to the technology, their opinions about its usefulness to success in the course, and 
their learning styles. Out of a course enrollment of 210, a total of 105 usable responses were 
obtained. The responses are summarized in Table 3. The responses were given on a 7-part 
Likert scale with 1 indicating a strong disagreement, 4 being neutral and 7 indicating strong 
agreement. In Table 3 responses 1 to 3 are aggregated as "disagree," 4 is neutral, and 5 to 7 
are aggregated as "agree." 



Table 3-Student Responses to Survey Statements (Web-based Section) 



Statement 


Disagree (%) 


Neutral (%) 


Agree(%) 


1 . ‘This class is increasing my knowledge 
and skills in computers.” 


39.0 


28.6 


32.4 


2. ‘This course has increased my use of 
e-mail.” 


45.7 


33.0 


21.3 


3. ‘This course has increased my use of 
the Internet." 


22.1 


25.0 


52.9 


4. “We have to use the computer too much in 56.8 
this class.” 


21.2 


22.2 


5. “Discussion is important to the learning 
process." 


4.9 


19.4 


75.7 


6. “1 feel 1 have more interaction with my 
classmates in this class than in 
other classes.” 


35.6 


22.1 


42.3 



The responses in Table 3, especially to items 1 and 2, must be interpreted in light of the fact 
that about 84 percent of the students own a computer (see Table 1). In spite of the high level 
of computer ownership, a majority of students indicated the course had increased their use of 
the Internet (item 3) and less than a fourth (somewhat more than the proportion who don't 
own computers) indicated that the computer was used too much in the class (item 4). Over 
75 percent indicated that the discussion associated with the course was important to learning 
(item 5). In item 6, in spite of the fact that the entire class meets only once per week, instead 
of three times, a larger percentage (42.3) indicated they had more interaction with classmates 
than those who indicated they had less interaction (35.6). 

In addition, Dr. Hardcastle administered an open-ended survey that asked students about the 
advantages and disadvantages of taking a "Cyber-course," a course that included an on-line, 
interactive component and required the use of a computer. The most frequently mentioned 
responses are shown in Table 4. 
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Table 4-Advantages and Disadvantages of a "Cyber-course 1 



Advantages 


% of Students Who Mentioned 


Better access to course materials 


29.2% 


Convenience/flexibility 


17.4% 


Improves interaction between teacher and student 


15.3% 


Disadvantages 


% of Students Who Mentioned 


Technical “glitches” 


32.5% 


Impersonal 


29.0% 


Lack of personal computers (access) 


10.5% 



Dr. Hardcastle is currently replicating the study using only data from sophomores in order to 
better control the level of study. In addition, she is beginning a longitudinal study to deter- 
mine the existence of any long-term effects, advantages or disadvantages, associated with 
this type of course. 

Conclusions Regarding Learning Outcomes in the Web-based Section of Philosophy 1204 

• The evidence regarding learning outcomes is somewhat ambiguous. Students in the 
web-based course scored better than those in the lecture section on eight of sixteen 
specific essay scoring criteria deemed important to philosophical discourse, there was 
no significant difference between the two sections on seven of the criteria, and the 
lecture section scored significantly better on only one criterion. Based upon these 
results, the web-based section's performance on the essay was superior. This result is 
consistent with the increased writing and interaction regarding the writing that was 
built into the course redesign, i.e., that increased practice and review should improve 
performance. Even though the web-based group showed superior performance based 
upon the sixteen criteria, letter grades assigned by the readers to this particular essay 
showed no significant difference between the two groups. 

• In terms of responses to the student survey, there is evidence that student behavior 
was affected to the extent that over half had increased their use of the Internet and 
that over three-quarters better appreciated discussion, peer review, and the value of 
informal writing as a component of the learning process (both of these results may 
indicate other learning outcomes that will be very important for students living in 
an increasingly digitized work and social environment). Less than a fourth of the 
student respondents felt there was too much use of the computer in the course. 

• Advantages most frequently mentioned were better access to course materials and 
convenience and flexibility in studying. The disadvantages most frequently mentioned 
were technical problems and the impersonal nature of the technology. 
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Introductory Microbiology 

Biology 2604 is a sophomore level introductory microbiology course that enrolls approximately 
200 students per semester. It is taught once each semester by Dr. G. William Claus who has 
been responsible for the course for several years. The class meets three times per week in a 
large lecture hall. There is an associated microbiology laboratory course, and, although it is 
not required, most students take the lab concurrently with the lecture course. 

Table 5 provides demographic data on the 185 students enrolled in microbiology in spring 
semester 1997. 



Table 5-Class Profile for Microbiology Students Spring 1997 



Class Level 


Percent 


Freshman 


4.1% 


Sophomore 


48.6 


Junior 


31.1 


Senior and Postbaccalaureate 


16.2 


Gender 


Male 


42.5% 


Female 


57.5 


Ethnicity 


Caucasian 


89.6% 


African-American 


2.8 


Asian 


4.1 


Latino 


0.7 


Native American 


0.7 


Other 


2.1 


Percent Owning Computer 


73.6% 



Course Transformation 

With the help of a grant from the Alfred P. Sloan Foundation, Dr. Claus and a development 
team in Virginia Tech's Educational Technologies Department restructured the course offer- 
ing. The course has been offered in this new format since spring semester 1996. The changes 
took the form of enhancements to the lecture presentation, network access to all course 
materials, and new communications capabilities. With design and technical support from 
the development team, all presentation materials (acetate overhead transparencies) were 
converted to digital format (color PowerPoint slides) which Dr. Claus used in lecture. These 
materials were in turn converted to files that students could access from the web site and 
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view or print out. An elaborate course web site was created that allowed students to access 
class announcements, frequently asked questions, course documents (e.g., syllabi), color 
slides, printable black and white slides, practice exams, live communication links to the 
professor and the teaching assistant (electronic office hours - "chat room"), direct e-mail to 
faculty, asynchronous discussions with other students ("message boards"), and links to other 
web sites related to biology and microbiology. In addition to the usual in-person contact, 
students were encouraged to contact Dr. Claus by e-mail and during the on-line electronic 
office hours. 

The Biology Department does not, as a general rule because of budget reductions, assign 
teaching assistants for their faculty who teach large, lower-level lecture classes. For this 
reason, biology faculty have a practice of evaluating students based upon machine-graded 
tests given two or three times per semester. This is in contrast to philosophy (and other 
high enrollment humanities and social science courses) which rely on more labor-intensive 
evaluation methods and therefore have three or four graders per course. 

Learning Outcomes: Grades 

Extensive grade data were available back to 1994, but it should be pointed out that Dr. Claus's 
goal was not so much to improve grades as it was (1) to improve student access to course 
materials and information and (2) to improve communication with faculty and among 
students. These course improvements probably do not influence grades so much as other 
factors, such as the changing qualifications of the student body, changes in student 
incentives, and changes in the examinations themselves. 

There were no significant grade differences among the years. As illustrated in Chart I, the 
mean grade fell from a high of 76.6 in 1994 to a low of 73.6 in 1995, then rebounded slightly 
to 74.0 in 1996 and 74.5 in 1997. 

Chart 1-Grades by Year 
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The distribution of grades 1994-1997 for Dr. Claus's course is shown in Table 6. Since the new 
version of the course was first offered in spring 1996, the last two years of data represent the 
period after the change occurred. There are no consistent differences from year to year. (A 
more valid comparison of grades would have to control for the characteristics of the students 
enrolled and the continuous change in the content of the course and the exam questions.) 
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Table 6-Grade Distributions for Microbiology 



Year 


A (%) 


B (%) 


C(%) 


D (%) 


F (%) 


1994 


20.3 


32.0 


23.4 


16.8 


7.4 


1995 


15.0 


29.9 


26.0 


17.4 


11.7 


1996 


18.3 


25.5 


24.0 


19.0 


13.3 


1997 


16.8 


29.7 


27.0 


15.7 


10.8 



Discussion: Grade Comparisons 

There are several reasons why grade comparisons do not make a good basis for comparison: 

(1) No standard tests are administered, either between sections taught by different faculty or 
between classes taught in different semesters by the same faculty. For example, Dr. Claus and 
others tend to change their tests and exams based on changes in teaching methods, the 
introduction of new content, identifying poorly worded questions, and many other factors, 
both conscious and unconscious. 

(2) Many of the benefits from this type of course transformation are practical ones, such as 
ease of access to materials, and these types of benefits do not necessarily translate into in- 
creased learning as reflected in test scores. If microbiology students save time because they 
can access materials faster and "anytime, anywhere," they do not necessarily use that extra 
time to raise their grade in microbiology; they may, for example, use it to maintain their 
grade in another course that lacks such efficiencies and therefore requires more time, or for a 
host of other purposes (e.g., other activities, recreation, catching up on sleep, etc.). 

(3) Test grades may not necessarily reflect the changes in the type of learning that students 
gain from the technological enhancement of instruction. For example, the microbiology course 
has improved graphics, which students can access at any time over the campus network. 
However, tests and exams — as in years past — are text-based and multiple choice. It could be 
questioned whether the limitations of these tests can reflect any increase in visual learning 
experienced by students. 

On the other hand, Dr. Claus states that he has been very careful and consistent in the design 
and writing of his test items. Although he has been constrained by class size (and the general 
lack of a teaching assistant) to using machine-graded multiple-choice tests and exams, he 
works very hard to write test items that force students to think about the subject and engage 
in problem-solving that requires them to reason with the scientific concepts and factual in- 
formation. Having taught some version of this course for almost 30 years, he has assembled a 
large bank of test questions that he has revised and refined over that period. Although this 
pool of questions undergoes constant revision, he is satisfied that the pool covers stu- 
dents' mastery of the material. He is relatively confident, therefore, that year-to-year grade 
comparisons have some validity. 

Student Surveys 

A series of surveys were administered to the microbiology class during spring 1997. Several of 
the questions sought student reactions to the technology and their opinions about its useful- 
ness to their success in the course. Approximately 145 usable responses were obtained from 
the 185 students enrolled in the survey given in the eleventh week of classes. A summary of 
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responses to these survey questions is provided in Table 7. The responses were given on a 
Likert scale with 1 indicating strong disagreement, 4 being neutral, and 7 indicating strong 
agreement. In the table, responses 1 to 3 are aggregated as "disagree," 4 is neutral, and 5 to 7 
are aggregated as "agree." 



Table 7-Microbiology Student Responses to Survey Statements 



Statement 


% Disagree 


% Neutral 


% Agree 


1 . ‘The on-line course materials focus my 
attention on what to study.” 


15.3 


9.5 


74.5 


2. The use of technology in this class creates 
a friendly, open environment.” 


22.6 


38.7 


38.7 


3. “Bringing the printed black and white slides 
to class is helpful [to learning].” 


4.4 


8.0 


87.6 


4. “Coming to class is less Important in this 
course than my other courses because we 
have on-line materials.” 


54.4 


18.4 


27.1 


5. The on-line discussion questions help 
me think more about the class material.” 


30.0 


21.9 


48.1 


6. “1 enjoy answering and responding to the 
discussion questions.” 


59.8 


17.5 


22.7 


7. The discussion questions show me real-world 
applications of biology.” 


28.6 


29.4 


41.9 


8. “On-line practice exams do not help me 
prepare for tests.” 


82.5 


11.7 


5.8 


9. “A class web page is not a good way to 
communicate and give out information.” 


81.7 


10.2 


8.0 


10. “Using the class web page saves me time.” 


22.0 


22.6 


55.5 



Another indication of how students view the changes that have been made in the course 
structure — technological or otherwise — is to ask them about the usefulness of specific changes. 
Students were asked to respond to the following question: "Suppose a fellow student asked 
your advice about taking this course. Using a 1 to 7 scale (where 1 is not at all useful and 7 is 
extremely useful), rate the following items on their importance and usefulness for success in 
the class." The responses are summarized in Table 8 where responses 1 to 3 are aggregated to 
"not useful," 4 is neutral, and 5 to 7 are aggregated to "useful." 
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Table 8-Microbiology Student Responses to Survey Statements Regarding Usefulness 



Statements Regarding Usefulness: 


Not Useful (%) 


Neutral (%) 


Useful (%) 


1 . printing out or getting the slides 
before class 


5.7% 


5.8% 


88.4% 


2. printing and studying the black and 
white lecture slides 


2.4 


3.3 


94.3 


3. studying the full color lecture slides 
on the computer 


66.2 


15.3 


18.6 


4. attending the lecture 


13.8 


11.4 


74.8 


5. reading the textbook 


28.2 


19.4 


52.5 


6. the class chat room sessions 


79.0 


13.7 


7.2 


7. the practice exams 


16.4 


10.7 


72.9 


8. owning a computer with access to 
the network 


17.9 


12.1 


67.7 


9. e-mail access to the professor 


30.7 


29.0 


40.4 


10. the Explore Section of the course 
web page 


72.6 


13.7 


13.7 


1 1 . taking the microbiology lab 
simultaneously with the 
microbiology lecture class 


20.2 


12.1 


67.7 



The responses to items 1 and 2 in Table 7 provide an important summary and in general are 
quite positive, but also illustrate the existence of some reservations about the on-line materi- 
als. Almost 75 percent of the respondents indicated the on-line materials helped them focus 
their study efforts but less than 40 percent felt the use of technology in this class added to the 
creation of a friendly open environment. Students liked access to the black and white slides 
before class and for study purposes (Table 7, item 3; Table 8, items 1 and 2). They also liked 
access to the practice exams (Table 7, item 8; Table 8, item 7). 

The web site has become an integral part of the class structure. Over 81 percent of the 
students indicated that they thought the web page is a good way to communicate and give 
out information (Table 7, item 9). To this point, however, students indicate only a modest 
interest in using e-mail to communicate with the professor (Table 8, item 9). Similarly they 
tended not to use the site for real-time communication. For example, Dr. Claus instituted 
"electronic office hours," when he or a teaching assistant were on-line in "chat" mode for 
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1 hour per day, 5 days per week, but few students contacted the faculty while they were on- 
line. Interviews with students revealed, however, that many read others' comments from 
the chat rooms and sometimes submitted queries to that page throughout the week, thus 
using it in an asynchronous mode. 

Dr. Claus also incorporated on-line discussion questions for grade credit. Students read and 
responded to broad-based, subject-related questions; after posting their response, they read 
the responses of other students in their chat group and then resubmitted their responses. 
Submissions were graded by undergraduate assistants. Student response to the discussion 
questions was somewhat less than enthusiastic, 23 percent indicated they enjoyed the discus- 
sion questions (Table 7, item 5), the chat room sessions were rated not useful by 79 percent of 
the students (Table 8, item 6). However, 48 percent did admit the questions helped them to 
think more about the class material and 42 percent indicated the questions showed them real 
world applications of biology (Table 7, items 5 and 7). 

Finally, it is worth noting that over 55 percent of the students indicated that the web page 
saved them time. 

The changes in microbiology were not initially intended to convert it to a distance learning 
environment. It was critical for the students to attend lectures because the online materials in 
the early version of the course contained only the presentation slides used in the lectures. 
(Students realize this — nearly 75 percent indicated that attending the lecture was useful for 
success in the course, see Table 8, item 4.) Subsequently, Dr. Claus has created a totally on-line 
version that is used by both distant and local students. 

Nevertheless, ready access to class materials and information outside of class was one of 
Dr. Claus's primary goals in transforming the course. Even on a resident campus, such as 
Virginia Tech, it can be frustrating for students to trek across campus and wait in line at a 
professor's office to ask a single question, to wait at the library during peak times to get 
access to reserve materials or practice tests, and most of all, to get notes for a lecture they 
missed. Similarly, faculty waste valuable time by answering the same question endless 
times and they are frustrated when they must get important information out to hundreds 
of students on short notice. Based upon the responses to items 9 and 10 in Table 7, students 
substantially agree that the web site has worked for these purposes. 

An additional indicator of the usefulness of the class web site is how frequently students 
access it. Table 9 shows student survey responses regarding how often they used specific parts 
of the web site. 
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Table 9-Student Responses Regarding Use of the Web Site 



Part of Web Site 


Never or 


Couple times a 


One or more 


One or more 


Accessed 


Only Once 


semester, but no 


times per 


times per 




(%) 


more than every 
other week (%) 


week (%) 


day (%) 


Class News Page 


0.8% 


8.8% 


64.5% 


25.9% 


Full color class 


73.2 


21.2 


5.7 




lecture slides 


Printable black and 


6.5 


45.2 


46.8 


1.6 


white slides 


Explore Page 


69.4 


29.0 


1.6 





Note the disparity in frequency with which students access different parts of the web site. 
When a part of the web site is perceived as useful, it is used heavily; if not, it is virtually 
ignored. For example, the news and announcements page communicated a change of test 
location or time, correction of errors in the slides, and other items of immediate interest. 
Furthermore, Dr. Claus estimated that he recovered from 5 to 10 minutes (10-20 percent) of 
each 50-minute class period for teaching purposes by using the news page instead of making 
announcements in class. He also was confident that he had reached everyone, including late 
arrivals and non-attendees. Likewise, the printable slides allowed students to print up-to-date 
class notes ahead of class, and then listen to the lecture instead of transcribe notes from the 
screen. In interviews with students, the capability to get notes before class was invariably 
mentioned as the greatest advantage of this technology. 

On the other hand, the color version of these slides was perceived as superfluous. In interviews, 
students indicated these slides were slow to download and could only be viewed on the computer 
(hard copies required access to a color printer). Similarly, few students accessed the Explore 
page, because no assignments were tied directly to it (extensive use of the Explore page would 
have required a substantial time commitment for students to go beyond the basic requirements 
of the course). Although it is not reported in the table, practice exams were also available at the 
web site and were heavily used before examinations. 

Changes in Student Attitudes 

In the fourth survey of the semester students were asked: "As a result of taking this course, 
please indicate how your attitude has changed toward the topics listed. Responses were pro- 
vided on a seven-part scale where 1 was most negative, 4 was neutral, and 7 was most positive. 
In Table 10 responses 1 to 3 are aggregated as "more negative," 5 to 7 as "more positive." 
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Table 10-Changes in Student Attitudes After Taking the Microbiology Course 



Topic 


More Negative 


No Change 


More Positive 


1 . Biology as a career choice 


8.7% 


34.3% 


56.9% 


2. Science as a field of study 


6.6 


26.3 


67.2 


3. The way science is taught in college 


7.4 


30.7 


62.1 


4. The way science is taught in 


26.3 


51.8 


21.8 


high school 


5. Multiple choice tests as a way 


19.6 


40.4 


39.7 


of showing what I’ve learned 


6. Attending the lecture 


15.3 


34.3 


50.3 


7. Usefulness of technology in 


7.4 


8.1 


84.5 


higher education (i.e., computers, 
multimedia, Internet) 



The data in Table 10 and Chart II provide a clear indication that student attitudes underwent a 
positive change during the semester. For items 1 through 6 it is difficult to determine whether 
the changes should be attributed to the learning process in general, to the introduction of the 
computer technology, to the course restructuring, to the instructor's personality, or to other 
factors. The students felt that the course had a positive influence on their choice of biology as 
a career field, on science as a field of study, and the way science is taught in college (as opposed 
to high school). The importance of such a boost in motivation cannot be underestimated dur- 
ing the crucial sophomore and junior years, when many students tend to drop out of the 
sciences. It is also encouraging that, despite the class size and the required use of multiple- 
choice tests, students seemed to be more positively than negatively disposed toward Dr. Claus's 
creative use of thought-provoking questions on his tests and to the lecture mode of delivery. 

Chart 11-Changes in Student Attitudes After Taking the Microbiology Course 
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The response to item 7 does provide good support for the usefulness of electronic media in 
higher education. Almost 85 percent of the respondents indicated that their attitude had 
become more positive toward such use of computer technology as a consequence of having 
taken the course. Such a result suggests strongly the students found the technology to be 
truly useful to them in their studies. 

Conclusions About Learning Outcomes in Microbiology 

• Although a statistical test for differences in learning outcomes based upon course 
grade comparisons over time was not performed, the evidence strongly suggests there 
is no difference in either mean grades or the grade distributions for versions of the 
course offered before and after the course redesign. The grades themselves provide no 
evidence that learning outcomes have changed as a result of the introduction of the 
network technology. (It is worth reiterating that Dr. Claus's objective in the course 
transformation was not to change the grades but to improve communications and 
increase student access to course materials.) 

• The student survey responses clearly indicate that the course web site has improved 
student access to course materials, which was one of the primary objectives of the 
course redesign. 

• The survey responses indicate, in terms both of activities and attitude changes, that 
students are learning to use the computer and the Internet as tools to assist them in 
their studies. Such results will have direct application in their careers after graduation. 



Institutional Renewal 

The Instructional Development Initiative is a large-scale effort to invest in the faculty of 
Virginia Tech by providing them with the opportunity to rethink their teaching and to 
explore the potential of instructional technology for improving the effectiveness of the 
teaching-learning process. This initiative originated jointly from the Office of the Provost 
and the Office of the Vice President for Information Systems. It is based upon the realiza- 
tion that digital technology is causing profound changes in the way people communicate 
with each other and gain access to various types of information. These changes have impor- 
tant implications for the curriculum both in terms of how the technology applies in par- 
ticular disciplines and in terms of how courses can be taught. Students must be educated 
about these applications in order to function in society and the labor market. 

Virginia Tech has met this responsibility with the Instructional Development Initiative and 
the Faculty Development Institute (FDI). The FDI provides both opportunities and incentives 
for the faculty, the campus's most valuable resource, to learn more about the application of 
the technology as it applies to their teaching activities and to work with each other in sharing 
ideas and experiences about the instructional applications. 

This investment in human capital has been augmented by providing faculty appropriate desk-top 
computing equipment and network connections. It also includes a large-scale effort to provide 
network access, specialized computer labs, and support for students; classroom upgrades and 
equipment in classrooms are provided to make them suitable for computer presentation and 
network access. The FDI began with three pilot faculty workshops during the summer of 1993, 
and has continued with an increasing schedule of workshops since May 1994. By July 1997, over 
1,400 faculty (out of approximately 1,500) from all academic departments had participated in 
customized 3- and 4-day workshops that cover topics ranging from computer basics and e-mail to 
multimedia course design and development and using the World Wide Web as an instructional 
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medium. As part of the initiative, each participating faculty member is furnished with a high-end 
computer, a full suite of software and an office Ethernet connection. 

Many "graduates" of the FDI (including Drs. Claus and Hardcastle) have continued to develop 
their courses and contribute to an exciting atmosphere that encourages change and innova- 
tion. Dr. Claus, in particular, attributes a renewal in his teaching to his experience in the FDI 
and subsequent support from the Department of Educational Technologies, which manages 
the FDI workshops. 

The two courses reviewed above provide specific evidence of how the institutional renewal efforts 
of the campus are affecting the instructional process and improving students' understanding of 
the use of computers. 



Student Access 



As a primarily residential campus, Virginia Tech's emphasis upon the use of the technology in 
the Instructional Development Initiative has been to improve convenience and access to course 
materials for on-campus students rather than attempt to improve access for place-bound and 
relatively distant suburban and urban populations. The Initiative is broadly conceived, how- 
ever, and is providing the basis for an expansion of Virginia Tech's distance learning activities. 

Such improvements may be important for classroom instruction. Professor Claus estimates a 
10-20 percent saving in class time by reducing the need for announcements and course manage- 
ment issues in class. If this time is redirected toward more instructional content, the efficiency of 
the instructional process is substantially improved. Similarly, to the extent students can obtain 
class information, class assignments, and access to information resources generally more efficiently 
over a network, they experience time savings that can also be redirected to more study. 
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Cost Estimates 



The component costs associated with redesigning and restructuring the courses include 
costs related to the Instructional Development Initiative and the Faculty Development In- 
stitute. Some of the costs are one-time and some are continuing. Table 1 1 provides the basis 
for cost estimates related to faculty who participate in the FDI and who then convert one or 
more of their courses. The estimates are dependent upon a series of assumptions that are 
spelled out in the table. The one-time costs of course conversion are amortized over a five- 
year period. Once the courses are converted, it is assumed faculty will maintain them. The 
estimate of $2,340 cost per course per term is based upon a conversion funded by Virginia 
Tech's Instructional Services unit. Many faculty, including Dr. Hardcastle, have converted 
courses without special funding to do so. 



Table 1 1 -Estimated Faculty and Course Conversion Related Costs Philosophy and Biology 



Item 


Amount 


$/Course 


FDI cost 

if two courses are converted... 
if cost is amortized over... 


$2, 000/faculty 
$1 ,000/course 
5 years 


$200/course/year 


Faculty workstation costs 

if useful life is... 
annual station cost... 


$2, 500/station* 

4 years 

$625/station/year 




if annual maintenance cost is... 
total workstation cost is... 
number of courses are converted... 


$94/station/year“ 
$71 9/year 
2 courses 


$360/course/year 


Course conversion costs*** 

Assistance from Educational Technologies... 

Faculty stipend... 
Total conversion cost... 


$16,500 

$5,000 

$21,500 




if cost is amortized Over- 


5 years 


$4, 300/course/year 


Estimated costs per course 




$4, 860/course/year 






$2, 430/course/term 



* Includes software. 

** Calculated at 15 percent of station cost. 

“‘Source: Virginia Tech, “Access” report, July 1997, pp. 15-17. 
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The Instructional Development Initiative also includes computer access for students enrolled 
in the restructured courses. Estimates of the costs of providing this access are shown in Table 
12 along with the various assumptions that are necessary in order to convert workstation 
costs to costs per enrollee in a converted course. 



Table 12-Estlmated Costs of Providing Computer Access for Students 



Item 


$/Enrollee/ 

Amount Course 


Cost of student workstation 


$2,500 


if useful life is... 


4 years 


annual station cost... 


$625/year 


annual maintenance cost... 


$94/year 


Total workstation cost... 


$71 9/year 


If stations are available... 


50 hours/wk 


and student workload/course/week is... 


5 hrs/course/week 


one campus station accommodates the 
coursework of... 


10 enrollees/term 


If half of the workload* is accomplished on 
student owned equipment, one campus 
workstation accommodates the equivalent 

coursework of ... 


or 20/year 
20 enrollees/term 


Workstation cost per enrollee per course 


or 40 per year 

$17.98 



* This is a conservative estimate based upon the data presented in Tables 1 and 5 to 
the effect that over 70 percent of the students enrolled own computers. Student com- 
puter owners, however, may still make some use of campus computers. (Virginia Tech 
has just instituted a requirement that all entering freshmen own a computer with 
specified capabilities, including a network connection.) 

Table 13 combines the cost estimates from Tables 11 and 12 to show total restructuring 
costs on a per course and a per course enrollee basis. Consistent with the initial emphasis 
of the Instructional Development Initiative to convert courses that enrolled large num- 
bers of students, the costs are calculated on the basis of 100 and 200 enrollments per 
course section. 
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Table 13-Estimated Course Restructuring Costs for Alternative Course Enrollments 



Section Enrollments 


100 


200 


Estimated course related costs 
(from Table 11) 


$2,430 


$2,430 


Estimated student workstation costs 
(@ $17.98 from Table 12) 


$1,798 


$3,596 


Total restructuring costs per course offering 


$4,228 


$6,026 


Total restructuring costs per course enrollee 


$42.28 


$30.13 



In summary, the estimated costs of the restructured courses ranges between $4,200 and $6,000 
per course section for the larger enrollment courses. This converts to between $30 and $42 
per student enrollee. 

To put some perspective on these latter numbers, assume a student accumulates 132 units 
to graduate (44 three-unit courses). If 25 percent of these courses (11) were taken in the 
restructured format, the additional costs on a per graduating student basis would be in the 
range of $331 to $465. Alternatively, if the 25,000 students at Virginia Tech take, on aver- 
age, 9 courses per year, total course enrollments are 225,000. If 25 percent of these enroll- 
ments (56,250) were in restructured course sections, estimated annual cost to the campus 
(including amortized costs) would be in the range of $1.7 to $2.4 million. 
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